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Discussion Paper 


Seeing the Forest through the Trees: Program Context Assessment 


\ 


Introduction 


Webster’s Dictionary defines ‘contextualize’ as the act of putting an action in context, 
especially for the purposes of study. This paper explores this technique as it applies to the 
early stages of a value-for-money audit. By expanding on the discussion of entity 
environment presented in the Comprehensive Auditing Manual (CAM), this paper argues that 
a thorough analysis of the context of program delivery is essential to understanding it and to 
developing value-for-money issues of significance. it builds on the premise that much of 
what is observed about the way a program is managed can be explained by the program’s 
interaction with its environment. 


One of the more difficult tasks facing the value-for-money auditor is sorting through 
and assessing the relevance of the volumes of information obtained during the initial stages 
of an audit. This is particularly so for information which does not readily relate to the 
program’s management systems, but rather describes the ‘environment’ in which the 
program operates. While the CAM deals with gathering information on the environment of 
the audit entity, this paper proposes a more methodical way of identifying, organizing and 
analyzing this information so that a cohesive picture of the program’s inter-relationships with 
its environment emerges. 


In order to more fully explain how contextualizing works and to show how useful it 
can be in identifying audit issues, the paper draws on the experiences of the External Affairs 
team during the value-for-money audit of the Consular Services Program. The paper 
discusses the major issues raised by the audit as they relate to the program’s context. The 
full results of this audit are reported in Chapter 12 of the 1989 Annual Report of the Auditor 
General. 


Environments, Programs and Audits 
The conceptual basis for the argument that understanding a program’s environment 


is essential to understanding the program has its roots in the theory and analysis of 
systems. In theory, there are generally two kinds of systems, open and closed. An open 


system is one that engages in exchange interactions with its environment. Open systems 
are contrasted by closed systems which are able to function in isolation from their 
environment. Open systems - and all organizations are open systems - survive and grow 
through exchange transactions with the environment. Survival and growth do not result from 
exchanges made inside the organization, altho:igh these are important. Rather, 
organizational success ultimately results from exchange transactions with various groups 
outside the system. In this sense, government programs are indisputably open systems. 
They continually interact with, and are products of, their environments 


The environment of a government program consists of the set of conditions, 
influences, and forces external to the program that have relevance to its goals and tasks. 
(While it is sometimes difficult to decide what is inside and what is outside of an open 
system, a useful approach to determining the boundary is to classify something as external 
to the system if it is outside the system’s direct control.) Specifically, the environment is 
made up of legislation and government policy, other departmental programs and other 
federal departments. Beyond this, it consists of the Cabinet, Parliament, service recipients, 
special interest groups and, perhaps, foreign governments. All of these influences are 
beyond the scope of program systems, yet they help determine the range of actions 
permitted to management. These factors combin: to create a climate for public sector 
management where management systems largely result from or respond to external forces. 
The best example of the degree of influence is that program mandates and the resources 
needed to achieve them are externally provided. Both program’s reason-for-being and the 
resources that it uses are outside the control of even the most senior managers. While 
some influence can, of course, be exerted by management in the legislative and budgetary 
processes, in the end the decision is someone else’s. 


Another good example of the degree to which management systems are products of 
their environments is shown in the phenomena of ’red-tape’. While it is popularly assumed 
that ‘red-tape’ in government is a function of bureaucracy in general, the problem of 
excessive rule-making (and rule-following) is not necessarily endemic in large private 
organizations the way it is in large public ones. Business firms rival and surpass 
government programs in size and complexity but, as James Wilson points out in his book, 
Bureaucracy, they have not developed nearly the same number of rules for hiring and firing, 
purchasing and selling, and for administration in general. The difference is explained by 
observing that these additional constraints are not management rules, but political ones. 
They reflect the political decision that fairness is the most important consideration in 


government administration - and as such these rules are perfectly legitimate. Left to their 
own ends, however, program managers might not assume the same procedural burden. 
The point is that management systems in government programs are deeply influenced by 
external factors - in this case, administrative policy. , 


The CAM discusses program environments in the sections on the audit overview and 
the survey. It asks questions which should provide the auditor with a clear appreciation of 
the program’s overall relationship with the environment. The following discussion takes this 
idea a step further by proposing a more methodical way of assessing the environment. The 
idea of the environment is enlarged to one of ‘context’ which captures the elements of a 
program’s environment as well as how they interact with the program. Our goal is not so 
much to identify the specific elements in a program’s environment - although this is the first 
Step - but rather to understand how the program and the environment interact as a clue to 
the reasonableness and sufficiency of management systems. 


The Analysis of Program Context 


It is useful to begin with the terminology of program context analysis. The context of 
a program consists of elements and interactions. Elements are defined as the external 
conditions or groups that interact with the program. The legal basis for the program is an 
example of condition. Examples of groups in the environment are other programs in the 
same department, other departments, central agencies, the Cabinet, Parliament, other levels 
of government, private enterprise, and the program’s clients, customers or other 
stakeholders. 


Interactions are defined as the relationships of the program with these conditions or 
groups. For example, the nature of the relationship between a condition such as legislation 
and a program management is one where a requirement to take action within statutory limits 
is imposed. The legislation can be said to set the jurisdiction within which program activity 
is legitimate. For relationships with groups, program management may interact with other 
departments or provincial ministries where their activities overlap. In these cases, the nature 
of the interaction is normally described through some sort of formal or informal agreement. 
As another example, the program interacts with service recipients through its delivery 
systems; the nature of this relationship is the effect that it has on the recipients. 


Identification of the Elements 


As required by CAM, the elements of a program’s context should be identified during 
the overview phase of a VFM audit (consit OAG Library Services and the departmental 
library for sources of context information - a literature search is always worthwhile). First, 
the auditor should develop a list of all relevant authoriies, including international treaties or 
agreements for which the program has complete or partial responsibility. Next. program 
clients should be identified as well as other federal programs acting in the same jurisdiction. 
Other departmental programs should be reviewed and understood. Provincial and, if 
appropriate, municipal responsibilities and activities should be determined. The auditor 
should also identify what kind of program is being audited. For example, the program may 
be regulatory or it may administer grants and contributions. Different types of programs 
have different contexts and it is reasonable to assume, at least initially, that programs of the 
same basic type should have similar management systems and practices. Where there is 
not so, it is important to understand why. Finally, the auditor snould describe the objectives 
that the program is supposed to achieve or maintain as precisely as possible. 


Case Study: Consular Services. | The Consular Services Program is responsible for 
providing protection and assistance to Canadians living and travelling abroad through 
the network of Canadian posts (embassies, high commissions, commissions, etc.). 
Within the Department of External Affairs, consular services are the joint responsibility 
of the Consular and Immigration Affairs Bureau at headquarters and the 105 (at the 
time of the audit) posts abroad. Headquarters provides functional advice on specific 
cases to the posts where the delivery of consular services is carried out by consular 
officers and local staff, supported by a growing network of honorary consuls. 


Among many other factors, the-delivery of consular services is influenced by 
international law, the social structure and culture of each country, the number of 
Canadians living and travelling abroad, the number of Canadians of plural nationality, 
and the competition for resources within the department. A multilateral treaty, the 
Vienna Convention on Consular Relations, makes it possible for Canada and the 
other 150 signatories to provide assistance and protection services to their nationals 
abroad while respecting the sovereignty of other countries. 


Exhibit 1, the Program Context Model, presents the elements (conditions and 
groups) that we identified in the program’s context. Groups are identified by 
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rectangles, conditions by squares. Also included are some of the more important 
inter-relationships between certain groups and conditions which interact separately 
from the program. 


There were two important findings resulting from the initial work of identifying 
the elements of the program’s context and describing their interaction with the 
program which helped determine the emphasis of later examination work. First, 
neither the External Affairs Act or the Vienna Convention on Consular Relations 
provided a clear description of the range and extent of consular services, or, in other 
words, what the Department had to do for Canadians in trouble abroad. In terms of 
our contextual analysis, this meant that program authorities imposed few 
requirements or constraints on program management. Second, roughly half of the 
services provided by consular services were delivered on behalf of other programs in 
External Affairs or other federal government departments. For example, the consular 
program provided citizenship application services for the Secretary of State and 
passports for the Passport Bureau. Consequently, the management of these 
programs were ultimately accountable for many ‘consular’ services and were sure to 
be involved in the management of their delivery. As a result, management systems 
and practices were largely determined by the requirements of various pieces of 
legislation as well as the expectations of the management of other programs. 


Analysis of Interactions 


The analysis of program context focuses on how each of the program’s inter- 
relationships with these conditions and groups works. The goal of the analysis is to 
describe the way in which the program inter-acts with these elements as simply as possible. 
A simple description is a good test of whether there is a clear (therefore significant) 
relationship, and whether it is well understood. 


Exhibit 2 is an extract from the context analysis worksheet for the Consular Services 
Program. The worksheet is a convenient way of organizing contextual information for future 
reference. Beginning on the left, conditions and groups in the program environment are 
listed. Next, the program’s interactions with these elements are descadad Any change in 


the relationship over a relevant period of time is recorded as well as the source of the 
information. 


Consular Services Case. As we began to understand the interaction of the 
Consular Services Program with its environment, it became clear that the program’s 
ability to meet its protection and assistance mandate was, to a great extent, 
dependent on elements over which program management had little or no control. 
The authority of a consular officer in a foreign country was severely limited and 
existed only to the extent that local authorities were aware of and abided by the 
Vienna Convention. What program managers, in Ottawa and abroad, could 
reasonably be held accountable for in this context was just as limited. 


The contextual analysis revealed several critical interactions affecting the 
delivery of consular services. Aside from the multilateral treaty which governs the 
provision of consular services around in the world, the most important condition 
affecting the Consular Services Program was the social environment of the various 
countries where Canadians live or travel. The needs of Canadians abroad and the 
demands placed on the consular services vary with the social and cultural differences 
between Canada and other countries. By identifying this basic interaction with the 
environment, we understood that the Department should not provide the same range 
and amount of services at each of its missions. It also meant that the rigor of 
management control over the delivery of these services varied among different 
countries. The arrest of a Canadian, for example, might reasonably warrant a 
perfunctory consular visit in one country, the immediate notification and involvement 
of Ottawa in another, or no action at all in a third. A more rigid service standard 
might mean that Canadians in some countries would have been receiving help they 
didn’t really need, while those in other countries did not receive enough. We shifted 
the focus of the audit to the more fundamental issue which was how management 
exercised control in spite of the unusual demands of a complex context. 


Exhibit 2 
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Inter-relationships Over Time 


An important aspect of understanding program context is the evolution of inter- 
relationship over time. Programs evolve in response to their environments and so here we 
are interested in history. Among the matters which may be examined is the evolution of 
program authorities. If possible, changes over time in the socio-economic conditions 
attributable to the program should be determined. Inter-departmental history - who did what 
and when - should be explored as well as trends in program resources in relation to other 
programs in the same department. Changes over time in management strategies, delivery 
mechanisms, and program policy should be explored and understood. 


Consular Services. The Consular Services Program dates back to the creation of the 
Department of External Affairs in 1905 as weil as the traditional government function 
of helping nationals abroad. However interesting this history was, we didn’t go back 
quite that far. First, we traced resources and workload levels back to 1975. In this 
way we were able to show that over time resources had decreased while the 
numbers of consular cases increased. Second, we also examined departmental 
consular policies of 10 years ago in comparison to current ones with a particular 
emphasis on the clarity of the guidance provided to officers in the field. Third, a 
dated, but useful policy document reviewed the problems with the Department’s 
approach to service delivery during the 1970s - many of which still existed. Finally, a 
program evaluation provided interesting information, not the least of which was the 
recommendation that the department should tailor its services to the conditions at 
each post. 


We developed the working hypothesis (to test during the examination) that 
management’s response to the program context was to exercise very little direct 
control over service delivery abroad, leaving key decisions in the hands of staff at the 
posts who, according to management, best understood local conditions. If this were 
so, then post staff should have access to policy statements which outlined the 
minimum obligations (responsibilities and service standards) to Canadians abroad as 
well as the areas left to the discretion of the posts for action on a case by case 
basis. In addition, consular staff should be appropriately trained or experienced, at a 
minimum at those officers assigned to posts where consular work was inordinately 
demanding. Failing these conditions, then there might have been deficiencies in 
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management systems and practices, possibly leading to failures in meeting service 
goals. 


Program Context and Audit Issues 


Congruency 


Once the program context has been established it serves as a framework against 
which to assess other information gathered about the program during the overview phase of 
the audit. The sum of the program’s inter-relationships gives the auditor a clear idea of how 
well the program fits in its environment. Do management systems and practices make 
sense given the context of the program?. Possible weaknesses in planning, program 
delivery, management controls and performance information can be more reasonably 
assessed from the point of view of the ’big picture’. A ’good fit’ is necessary for the 
program to work well. 


Consular Services. Once we had identified the context of consular services delivery, 
we assessed the degree to which program management systems and practices fit 
with the environment. We were specifically interested in how reasonable it was to 
leave so much discretion in the hands of staff at the posts with little management 
monitoring. 


The Department had produced general consular policies which identified the 
protection and assistance services that its staff may provide to Canadians abroad. 
These guidelines were meant to allow consular staff the flexibility to adapt 
departmental services to the circumstances in other countries. Program management 
relied on the judgement of consular staff in interpreting these policies to respond 
appropriately to the situations of Canadians needing help. 


Since consular services is a program unique to the federal government, we 
did a comparative analysis of the consular programs of three other countries. We 
hoped that the results of this analysis would help us determine whether what we were 
observing at External Affairs made sense. As it turned out, the other countries’ 
programs were managed in similar ways, with an emphasis on a definition of 
minimum responsibilities and local standards of service. With this finding we were 
able to establish a point of comparison to the management approach at External 
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Affairs. By the end of the audit we had successfully argued that good management 
practices in this context dictated that the Department should specify for each post the 
range and extent of consular services that would be provided in response to local 
conditions, supplemented by the adequate training of consular staff. The effort that 


we put into analyzing the context of the program during the audit formed the basis 
for this argument. 


Risk and Significance 


By placing the program in its context, we are better able to assess the risk and 
Significance of failures to achieve program results. The analysis of risk should consider the 
impact of program context on the achievement of program results, and the ultimate impact 
on clients. Assessments of significance, or the impact of program failures on its clients and 
the degree to which these failures would be of interest to Parliament, should also reflect the 
mitigating influence of the program’s context on the impact. 


Consular Services. Given the uncertainty associated with the context of consular 
services and the degree to which program results were uncontrollable, the risk of 
service failure was very high. Compounding this situation was the significance placed 
by Members of Parliament on any misfortune that their constituents encountered while 
travelling abroad. Both Parliamentarians and the Department are sensitive to cases 
involving Canadians in trouble abroad and the media attention surrounding them. 
Other than local authorities, there were no other agencies which could be expected 
to compensate for program failures. All of these factors argued for a more explicit 
description of the range and extent of the services available to Canadians at each 
post abroad. 


We recognized that there was some need for flexibility and discretion at the 
posts. However, we also concluded that a very general description of consular 
responsibilities increased the risk that unnecessary services would be provided, or 
that required services would be neglected. Total reliance on the judgement of 
individual consular staff to properly interpret their responsibilities on a case-by-case 
basis increased the risk of inconsistency in the treatment of Canadians with similar 
needs. 


Seeing Inside Out 


An understanding of program context allows the auditor to see the program from the 
inside out, or from the manager’s point of view. This perspective has two important 
benefits. The auditor will be able to develop program-specific criteria earlier in the audit. 
Second, and more important, the auditor will be more likely to make recommendations that 
are relevant and practical to program management on issues that are likely to be significant 
to Parliament. 


Consular Services. Our understanding of consular services resulting from the 
contextual analysis enabled us to develop audit criteria that reflected the realities of 
service delivery outside of Canada. In addition, we had developed a thorough 
knowledge of the intricacies and challenges of consular work from the perspective of 
the consular officers and External Affairs’ neadquarters. For example, due to our 
early recognition that the range and extent of service should vary form post to post, 
the criteria, and the audit examination, focused on whether the Department had 
assessed by post what kind of help should be provided as a guide to its staff 
abroad. We looked at whether headquarters assessed the degree to which post 
responses to individual cases conformed to overall program objectives. We also 
looked at the variation in range and extent of services among posts and among 
cases at the same post. For human resources management, we looked at how well 
prepared consular staff were when they are assigned abroad given the variety of 
foreign cultures with which they were expected to deal. We also examined the effort 
that program management put into the maintenance of relationships with those 
groups in the environment that could help with service delivery, such as the travel 
industry, the foreign services of friendly powers and foreign police forces. 


Conclusion 


This paper had modest intentions. First, | hoped that it would help other auditors 
develop new kinds of questions about how programs work and why they work that way, and 
to encourage them not to be satisfied with ‘that’s-the-way-it-has-to-be’ explanations. To do 
this, I've suggested a systematic approach to what every auditor already does intuitively. 


And second, | thought it might be more interesting if | drew on the experience of the 
External Affairs team during the VFM audit of Consular Services since program context was 
the key to the audit issues. Something that | think is missing in the Office is a discussion of 
how the teams developed the audit issues described in the Annual Report and so that is 
what I’ve tried to do with this paper. 
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